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Abstract. We consider the problem of supporting rank and select op- 
erations on a bit vector of length m with n 1 bits. The problem is con- 
sidered in the succinct index model, where the bit vector is stored in 
"read-only" memory and an additional data structure, called the index 
is created during pre-processing to help answer the above queries. We 
give asymptotically optimal density-sensitive trade-offs, involving both 
m and n, that relate the size of the index to the number of accesses to 
the bit vector (and processing time) needed to answer the above queries. 
The results are particularly interesting for the case where n — o(m). 

1 Introduction 

We consider the problem of representing a bit vector S[l..m] of length m, and 
supporting the following operations, for x £ {0, 1}: 

— rank^S', i) returns the number of occurrences of x in the prefix S[l..i]. 

— se\ect x (S, i) returns the position of the ith occurrence of x in S. 

Such a data structure is called a fully indexable dictionary (FID) [25]. We con- 
sider this problem in the context of systematic encodings, also known as the 
succinct index model. In this model, the bit vector S is not directly accessible 
to the data structure as "bits in memory" : instead, when answering a query, 
the data structure can inspect parts of S through an access operation, which 
may be relatively expensive. In order to reduce the number of access operations, 
we augment S with an index, or a data structure that contains pre-computed 
information specific to S. To answer a query, the data structure performs a com- 
bination of access operations and "local" computations using the index as well as 
results of the access operations. We consider three measures of the performance 
of the data structure: 

* Some of these results were published in preliminary form in the proceedings of 
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Springer. Orlandi's work was done while affiliated with the University of Pisa. 



i. The size of the index, also termed the redundancy, 

ii. the number of access operations, 

iii. the amount of "local" computation. 

(We do not include preprocessing time, or access operations performed dur- 
ing preprocessing.) We provide new and tight density-sensitive lower and upper 
bounds for this problem, where the above parameters depend both upon the 
length m and the weight n of the bit vector, where the weight is the number of 
Is in S. One can, without loss of generality, assume that n < to/2; our primary 
interest is bit vectors of relatively small weight, i.e., the case n = o(m). 

We also mention the non-systematic model, as our succinct indices will in fact 
consist of a number of non-systematic FIDs. In this model, the string S is explic- 
itly given as input. The algorithm is responsible for storing S in an information- 
theoretic minimum amount of space, according to one of a number of possible 
measures of the "information content" of S. In this setting, the redundancy is the 
space usage of the data structure above and beyond the information-theoretic 
minimum amount needed to store S. 



1.1 Motivation 

Representing a bit vector to support rank x and select^ is one of the most funda- 
mental operations in the field of space-efficient data structures [24] . Solutions for 
this problem (such as FIDs) are used in text indexing [15, 16, 6] and representing 
semi-structured data [9,5], for example. Considering systematic data structures 
for this problem appears to have been initiated by the desire to prove lower 
bounds on index size [8]. However, as the field developed, both algorithmic ad- 
vantages and technological motivations. Service based computing [?] and remote 
data storage, two of the main ingredients of nowadays is often referred to as "the 
cloud", have made the case for systematic data structures stronger. In papers 
such as [7, 26, 14], the first steps towards recognizing the algorithmic advantages 
of this approach were taken, and the succinct index model was fully formalized 
(from an upper bound perspective) in [1]. By decoupling the representation of 
S from the set of operations that are being supported, it is noted in [1] that a 
succinct index offers many advantages including optimal compressibility of the 
data being indexed and easy integration of different indices over the same data. 
Furthermore, a succinct index can also be used in cases when the data being 
indexed is never explicitly stored, but computed on demand (for example, see 
[12])- 

Finally, the situation that we emphasize here, namely bit vectors of length to 
with weight o(m), is very important in applications (both practical and algorith- 
mic). Such bit vectors are frequently encountered as the characteristic vectors of 
sparse sets that naturally occur in practical applications, see e.g. [17]. In addi- 
tion, many space-efficient data structures often need to represent sequences that 
have few Is by construction [15,9]. 
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1.2 Results - Old and New 



There have been a number of results on FIDs, and we do not describe all of 
them here. We note, however, that the redundancy of FIDs has been a focus of 
research in the non-systematic setting as well, and successive papers [2, 25, 13, 
21] have reduced the redundancy of Jacobson's original representation [18]. Very 
recently, matching non-systematic lower bounds have been shown [23]. 

Before we describe the systematic/succinct index results for FIDs, we first 
state the model more precisely. Recall that we want to support operations on 
a given bit vector S of length m, accessible through an access operation. We 
pre-process S (for free) and create an index / of size r bits. Subsequently: 

— For the lower bounds, we assume that access(z), for 1 < i < m, returns the 
i-th bit of S. When answering the query, the data structure can read / for 
free, and is only charged for access operations. This model is also referred to 
as the bit-probe model. 

— For the upper bounds, we take \i = [lgm] assume that access(i), for 1 < i < 
|"m//z|, returns the sequence of \i bits of S beginning at position [i — 

The "local" computation of the data structure on the index I is analyzed on 
the RAM model with word size O(lgm) bits. The performance of the data 
structure is measured both in terms of the number of access calls it makes as 
well as the number of operations performed during its "local" computation. 

The aim is to study the trade-off between r, the size of the index, and the cost of 
the operations as measured above. In the non-systematic model, upper bounds 
are also in the word RAM model and only the redundancy and the speed of 
operations is measured. 

We begin by noting that ranko(i) + ranki(i) = i, so a data structure needs to 
support only one of these operations, and refer to both as rank if this is otherwise 
immaterial. On the other hand, when we refer to select, it is simply as informal 
shorthand for referring to "select and/or selecti". 

We now summarize existing and new results. A number of recent results 
give lower bounds on the redundancy of systematic encodings [8, 19, 10]. It has 
been shown [19,10] that i?(mlglgm/lgm) redundancy is needed to support 
FID operations in O(l) time, matched by upper bounds in [10,25]. Hence, the 
redundancy of systematic FIDs appears to be a solved question. 

The lower and upper bounds, however, are not sensitive to the weight n of 
the bit vector. For example, when n = 1, it is easy to see that redundancy of 
O(lgm) bits suffices to support all operations in 0(1) time. As already noted, 
one often has to support FID operations on bit vectors that are constructed to 
have few Is. Thus, it is interesting to study the redundancy required to support 
FID operations as a function of both m and n. Previous density-sensitive lower 
bounds were provided by Golynski [10, Theorem 4.1], who showed that r = 
i7(nlglgm/lgTO) for t = O(lgm); as we see below, this lower bound is not 
optimal. Miltersen's [19] work implicitly contains an optimal lower bound for 
the case n — @(ra/lgm). 
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Worst case (over all n) 
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Table 1. Redundancy of systematic encodings of O(lgm) bit-probe FIDs. 



The lower bound is a complete trade-off that specifies the minimum redun- 
dancy required by any data structure that makes at most t bit-probes for any 
values of m,n and t. For simplicity, we focus on the case t = 0(\gm), and the 
new results are shown in Table 1. As can be seen, the previous lower bound of 
[10] matches the new ones only for n = 0(m) (i.e. for the dense case). For the 
case n = 0(m/ \gm) for instance, the new lower bound is £2(n) = fi{m/ lgm), 
whereas the old lower bound was Q{n lg lg mj lgm) = S7(m\g\gm/ (lgm) 2 ). 

We show matching upper bounds, giving succinct indices that support all 
FID operations, perform O(l) access operations and support queries in O(l) 
time if, additionally, m/n = (lgm) ' 1 ). Note that in the upper bound model, a 
single access operation reads lgm consecutive bits of S, so the lower and upper 
bounds are indeed comparable (indeed, the lower bound allows the algorithm to 
probe O(lgm) arbitrary bits per query, not just consecutive ones). Furthermore, 
the restriction in the upper bound that the "local" computation takes constant 
time only when m/n — (lgm) ' 1 ) is due to the lower bound on predecessor 
queries in the RAM model [22] (recall that the lower bound assumes that all 
"local" computation is for free). 

The rest of the paper is organized as follows. Section 2 contains the upper 
bound and begins with a (non-systematic) data structure that represents a bit 
vector of length m with weight n in B(m, n) + 0(min{n, m — n}) = 0(B(m, n)) 
bits and supports only selecto and selecti in 0(1) time, where B(m, n) = lg |~(™)] 
(provided min{n, m — n } = m/(\gm)°^). In fact, FIDs arc known that can 
achieve a space bound of B(m 7 n)(l + o(l)) bits [13,21], but they are signifi- 
cantly more complex; the experimental work of [20] suggests that our approach 
is practical. We then use this data structure in our systematic index. The lower 
bounds are described in Section 3 and are based upon the general choices tree 
framework of [10]. 

2 Upper Bounds 

2.1 Non-systematic FIDs for moderately-dense bit sequences 

We begin by giving a number of results on non-systematic FIDs; these FIDs 
will be heavily used in our succinct index. In this section We begin by stating a 
classical result on FIDs, due to Clark and Munro [2]: 

Lemma 1. There is a FID that stores a bit vector S of length m using m + o(m) 
bits and supports all operations in 0(1) time. 
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Next, we show the following lemma, which is essentially the Elias-Fano repre- 
sentation of a sequence [4], straightforwardly augmented with the rank operation 
in O(l) time (a related data structure was shown to have good practical perfor- 
mance in [20]): 

Lemma 2. We can store a bit vector of length to and weight n, providing that 
n > m/(lgm) c for some constant c > 0, such that we can support rank and 
selecti in 0(1) time using B(m,n) + 0(n) bits. 

Proof. Assume n < to/2 - if not, store S in Lemma 1. Let 1 < x\ < . . . < 
x n < to be the positions of the Is in the input bit sequence S, and equate S 
with the sequence {xi}. Let b = 2r i s(" l /™)l , and distribute S into z — [to/6] = 
0(n) buckets, each of length b by placing x <G S into the bucket L X /^J- We 
encode the number of elements mapped to each bucket in unary as the bit 
vector B = 01 bl 01 b2 0. . .01 6 *, where h is the number of l's in the i-th block. 
We store B, whose length is n + z, as a FID, and we store an array L where 
L[i] = Xi mod b. By Lemma 1, we need 0(n) bits to represent B, and since 
L requires n|~lg&] = nlg(m/n) + 0(n) bits, the entire data structure takes 
n \g(m/n) + 0(n) = B(m, n) + 0(n) bits. 

The bit vector B demarcates bucket boundaries in L, as the i-th bucket con- 
sists of the elements in L corresponding to the Is between the indices select (z) 
and selectn(z + 1) in B. If a bucket has more than ^- , lg , m elements in it, then we 

u V / 2c lg lg m ' 

store a fc-way search tree on top of the elements of L that belong to the bucket, 

for k = lglgm j (t ne leaves of this search tree are the elements of L and are 

not stored). Since each bucket has only 0(m/n) = (lgm)°^ elements in it, and 
each element in L is of size lg(TO./n) + O(l) = clglgm + O(l) bits, the search 
tree is of constant height, and the sum of the sizes of all the search trees over all 
the buckets is at most 0(n(lglgTO,) 2 /lgTO) = o(n) bits. 

To support selecti (i), we find the bucket to which the z-th 1 belongs, by 
finding the number of Os before the i-ih 1 in B using the FID for B. (Specifically, 
compute selecti (B, — i + The remaining lower-order bits are read from the 
location L[i\. To support rank(i), the i-th position belongs to the j-th block, 
where j = \ i/b\. We first find the number of l's upto the beginning of the j-th 
block (by finding the number of l's upto the j-th in B as select (£?, j) — j + 1. 
We then search for the key i mod b in the j-th bucket of L using the search tree 
structure stored (if any). Note that since the elements of L are O(lglgTO) bits 
long, all keys stored at an internal node of the tree can be stored in a single word, 
and a predecessor search at an internal node can be done in O(l) time using 
table lookup. If no search tree is stored, then there are no more than k elements 
in the bucket, and one can perform table lookup using the entire bucket. □ 

The following lemma is key in what follows. Although it is not substantially new 
(the key ideas are adapted from [3, 13]) the form of the lemma is particularly 
convenient for what follows. 

Lemma 3. Given integers N , Ni > and M = N a +Ni, such that min{A , Ni} > 
Mj (lg M) c for some constant c, we can store a bit vector S with n < N Os 
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and n\ < Ni Is, using B(M,N\) + O(min{N , Ni}) bits, such that selecto and 
selecti are supported in 0(1) time. 

Proof. Assume that S begins with a 1 and ends with a (otherwise adjust 
Proposition 1 below appropriately) . We describe S by two bit-vectors Rq and Ri , 
defined as follows. If there are r runs of Os of length l\, I2, ■ . ■ , l r in S, then i? 
is simply Zl ~ 1 10' 2_1 l . . . 0' r_1 l. R\ is defined analogously, using the runs of Is 
(note that there are r runs of Is as well). 

Proposition 1 ([3]). Let S be as above. Then: 

— Rq and R\ are of length uq and n\ and both have weight at most r. 

— selecti (S, i) = selecti (RO, ranki(i?l, i — 1)) + i 

— selecto(5, i) = selecti (i?l, ranki(i?0, i — 1) + 1) + i, 

taking selecti(-,0) = ranki(-,0) = on the RHS in the last two bullet points. 

Thus, we only need to support selecti and ranki on i? and R\. If N < N\, 
then store i?o according to Lemma 1, which takes n + o(n ) = O(N ) bits, and 
store i?i using Lemma 2. We pad out R\ to have exactly N\ Is and Nq Os by 
appending N\ — m Is and N n — r Os (note that r < No), and represent R\ using 
Lemma 2 (the padding is done to satisfy the preconditions of the lemma) . The 
space bound is B(M,N Q ) + O(N ) = B{M,N 1 ) + 0(min{iVo, Ni}) bits in this 
case . If Ni < N , switch the representations of R and R\, the space bound is 
still the same. □ 



2.2 A succinct index for FID operations on sparse bit vectors 

Recall that a systematic encoding of a bit vector S accesses S through access 
operations, building an index to minimize the calls to access and to support 
the operations rapidly. We begin by stating the target we are aiming for - the 
function RQ defined in the next proposition is in fact the same function as 
that of Theorem 2, with the difference that the lower bound is in the bit-probe 
model, and the upper bound assumes that each access operation returns 0(lg m) 
consecutive bits of S. We then use the data structures developed in the previous 
section to create succinct indices for rank and selecti, and then for selecto. 

Proposition 2. Let m > be an integer, and let n be some integer function of 
m, where 1 < n < m. Let t > be an integer and let ji = \t\gm\. Then, if 



R(m, n, t) 



tlgm 



1 g( M ^) ,ifn = w(m/(ilgm)) 
n ( 1 + l s(^)) ,ifn = 0(m/(tlgm)) ) 

then B(n + [m//z] , n) = 0(R(m, n, t)). 

Proof. Follows from the standard approximation to the binomial coefficients, 
namely lg (£) = 0(61g(o/6)) if b < a/2. 
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A succinct index for rank/selecti 

Lemma 4. Given a bit vector S of length m with weight n, where min{n, m — 
n} > mj (lgm) c , for some constant c > 0, there is a succinct index that supports 
rank and selecti on S that uses 0(t) time, 0(t) access calls, and 0(R(m,n 1 t)) 
bits of space, for any t — (lgm) ' 1 '. 

Proof. Partition S into contiguous blocks of size fj, = \t\gm] each, and let 
Hi > denote the number of Is in the i-th block. We represent the sequence 
OD = l ni 01" 2 . . ., which has n Is and \m/fi\ 0s using Lemma 3; the index size 
is B(n + [m///| ,n) + 0(min{m/^, n}) = 0(R(m,n,t)) bits, as required. Using 
standard approaches, we can assume that the "local" computation needed to 
perform rank and selecti operations on a block can be done in 0(t) time using 
pre-computed lookup tables of size 0(m 2 / 3 ). 

— To compute rank(S*, i), let j = [i/n\- If j = 0, then the answer is ob- 
tained by reading the first block of S with 0(t) access operations. Otherwise, 
selecto (OD, j) — j gives the number of Is in blocks 1, . . . , j, and reading the 
next block with with 0(t) access operations gives the answer to the query. 

— To compute selecti(S', i), we first compute j = selecti(OD, i) — i + 1, giving 
us the block in which the i-th 1 lies. A call to selecto(OL ) , j — 1) gives the 
number of Is in blocks 1, . . . ,j — 1, after which 0(t) access calls suffice to 
compute the answer. □ 

A succinct index for select 

Lemma 5. Given a bit vector S of length m with weight n, where min{n, m — 
n} > m/(lgm) c , for some constant c > 0, there is a succinct index that supports 
selecto on S that uses 0(t) time, 0(t) access calls, and 0(R(m,n,t)) bits of 
space, for any t = (lgm) ^). 

Proof. We divide S into blocks of size fi — [tlgm] as before. Let x\ < X2 < 
. . . < x z be the positions of the 0s in S such that ranko(5 l , xi) = for i = 
1, 2, . . . , z = [(m — ri)/ n\. Taking x = 0, if there are bi Is between Xi-i and 
Xi, the bit vector SP = l^Ol^O . . . l b *Q has at most n Is and mj[i 0's, and 
is represented using at most B(n + \m//i \ ,n) + 0(min{n, m/fj,}) bits using 
Lemma 3 so that select and selecti are supported in O(l) time. Observe that 
selecto (S, in) = i^ + selecto (SP, i), so we now are able to answer select queries 
for the positions i/i, for i = 1, 2, . . . , z. 

To answer general selecto queries, we proceed as follows. With each position 
Xi^ we associate the gap [x^, X( i+ i) M ). We say that position x ifl is the starting 
point of a long gap if aVj+iw — Xi^ + 1 > 2fi and define a set LG to be those 
positions which are the starting point of a long gap (see [2] for a related idea). A 
key property is that there are at most n/fi long gaps and that X)i M GLG x (i+i)^ ~ 
Xift + 1 — 0(n). This is because any long gap contains at least \x Is, and so there 
are at most n//z long gaps; but each long gap always contains /i 0s, and there 
are at most n Is that lie within long gaps. 
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The bit vector LG (overloading notation), whose i-th bit is 1 iff is the 
starting point of a long gap, has z < (m — nj/fi Os and at most n/fi Is. We 
represent LG using Lemma 2 (if LG has fewer than nf/j, Is we append Is to the 
end, so that LG is not too sparse to apply the lemma). The space used by LG 
is B([m//i\ , [n//i\) + 0(m//x) bits, which is negligible. Observe that selecto(i) 
can be computed in 0(1) time if x^.yi/^ is not the starting point of a long gap 
(which can be tested using LG), as we can read all the bits in the gap starting at 
x n-U/n\ using 0(t) access operations, and operate on them in 0(t) table-lookups. 

We now consider selecto when the answer is in a long gap. Since there are 
at most n/ [I long gaps of total length 0(n) for some constant c > 0, the max- 
imum possible number of blocks b that the long gaps can straddle is at most 
0(n/fi) blocks in S. Furthermore, the maximum possible number t of 0s in 
long gaps is 0(n). If the zth block (partially or fully) contained in a long gap 
has Zi zeros in it then the bit vector ZD = Zl 10 Z2 l . . . Zt l is represented 
using Lemma 3. Observe that ZD has t 0s and b Is, so its space usage is 
0(B(t + b,b)) = 0((n/fi) lg/x). This is always 0(R(m, n,t)), since if n = oj(m/fi) 
then R(m,n,t) = 0((m//x) lg/x), and if n = 0(m//j) then R(m, n, t) — 0(n\gfj). 

The steps to answer select when the answer is in a long gap are as follows: 

(a) Let r = fi and obtain x r = selecto (S, r). 

(b) If x r is the starting point of a long gap, then q = ranki(LG, r/fi) gives the 
the number of long gaps preceding x r . 

(c) The number of block boundaries crossed by the interval from x r to selecto (S, i) 
can be obtained by taking the difference in position between 0s correspond- 
ing to these in ZD (which is se\ecto(ZD,i mod fi + q/i) — selecto (ZD, qfi)) 
and subtracting from it the number of 0s in ZD between these two positions 
(i mod ii— 1). Since we know the block in which x r lies, we have identifed the 
block in which selecto (S, i) lies; it now merely remains to find the number 
of 0s before this block, which is just a rank operation and can be answered 
by using the OD bit vector of Lemma 4, and then reading the block itself in 
0(t) access operations and 0(t) time. □ 

Lemmas 4 and 5 show the following main theorem: 

Theorem 1. Given a bit vector S of length m with weight n, where min{n, m — 
n} > m/ (lgm) c , for some constant c > 0, there is a succinct index that supports 
all FID operations on S that uses 0(t) time, 0(t) access calls, and 0(R(m,n,t)) 
bits of space, for any t = (lgra) ^ 1 ', where R(m,n,t) is as defined in Proposi- 
tion 2 

We remark that the condition that min{n, m — n} > m/(\gm) c is essential to 
get 0(1) time operations, as the predecessor lower bounds of [22] also apply in 
this setting. Given a set S C [to], l^l = n, wc can represent the characteristic 
vector of S in 0(n) words of space in the index by storing the positions of all 
Is in the bit vector in the index. Since R(m, n, t) is also at most O(n) words of 
memory, if we could achieve an index of size R(m,n,t) and support 0(l)-time 
rank operations on the characteristic vector of S for n = m/(lgm) w ( 1 ^ we would 
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be able to solve predecessor queries in 0(1) time for n = m/flgm)"^ 1 ) using 
0(n) words of memory, which is impossible [22]. 



3 Density-sensitive lower bounds 

In this section, we first develop new bounding techniques for binomial coefficients 
and use them to prove the following theorem. 

Theorem 2. The size of the index to support the operations ranki or selecti on 

bit vectors of length m and weight n satisfies 



Golynski [10] showed that r = Q{{n/t) \gt) for both ranki and selecti. This 
bound is tight only in the case of constant density bit vectors, i.e. when n = 
0(m). For sparse bit vectors, e.g. when n < m/t, the bound of [10] is smaller 
than optimal by almost a factor of t. 

In this section, we refine the techniques used in [10] and show tight bounds 
on the index size for rank and select operations in systematic encodings. We 
prove bounds for the rank problem, and defer the details of select to the full 
version. Consider 7 queries Q* — { "ranki (m/7)", "ranki (2m/7)", .. .}, where 
7 is a parameter which will be chosen later such that 7 divides to. Let 1(B) 
denote the index of size r that is used by the ranki algorithm on B. Construct 
the decision tree T for the following procedure: first probe all the r bits stored 
in /, and then simulate the computation of Q* queries one by one. The nodes on 
the first r levels of this tree are labeled by u I[p] = ?" for 1 < p < r, and the rest 
of the nodes are labeled u B[p] = ?" for 1 < p < to. The edges are labeled by or 
1. Let a; be a leaf of T. For simplicity of presentation, we perform arbitrary extra 
probes, so that all the leaves of T are at the same depth r + t-f. Call B compatible 
with x if 1(B) corresponds to the first r edges on the path from the root to x, 
and the probes performed on B by our computation correspond to the rest of the 
edges on the path. The set of such vectors is denoted by C(x). We note that the 
bit vectors B\^B 2 G C(x) share some common features, e.g. I(B\) = /(-B2), the 
locations and the contents of the probed bits by our computation are identical, 
and the answers to the queries in Q* on B\ and B 2 are also identical. 

The idea of the lower bound proof is as follows. Consider the set % of (™) 
bit vectors of length m with n 1-bits. These bit vectors are distributed among 
the leaves in some fashion. Imagine, that we have a bound |C(a;)| < C*(x), and 
let C* be the sum of C*(x) across all the leaves. Being an upper bound on the 
number of leaves, C* is at least \H\. The bounds derived in [10] are such that 
C* = 2 r D* , where D* does not depend on r (intuitively, C* is proportional to 
the number of leaves in T). Hence r should be at least \g(\H\/ D*). 

The bound C*(x) can be derived as follows. Let us split all the locations in 
the bit vector into 7 blocks, the first block spanning positions 1,2,..., m/7, the 




W (l) 
0(1) 
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second block spanning positions m/7 + 1 , m/7 + 2, ... ,2m and so on. Let Ui (x) 
be the number of unprobed locations in the i-th block in the bit vectors that are 
compatible with x, yi{x) be the number of 1-probes performed on the block (on 
the root to leaf path), and Vi(x) = ranki((i + 1)771/7) — ranki(im/-f) — yi{x) be 
the number of unprobed 1-bits in the block (their locations can be different for 
different B G C(x), however the number is fixed for a given leaf, since both yi 
and the result of rank x queries are known). From now on, we omit parameter x 
and use just Ui, v i} yi to denote these quantities, e.g. define y := J2i Vi- We have, 

\c { x)\<c^) = h)M...( u A, a) 



where U := ^2 u i = rn — tj (since exactly t-y positions are probed for each 
leaf) and V :— X) w « = n ~ V ( smce V is the total number of probed 1-bits). 
Let L y be the group of leaves for which there are exactly y 1-probes. Note that 
\L y \ = 2 r (*J). For each y, let x y be the leaf in L y that maximizes the product (1). 
Hence, partitioning all 2 r+ * 7 leaves w.r.t. y, we have, 

min{t7,n} . . 

C* <2 r J2 ( 7 )C*(x y ) < n2 r X, 

y=0 ^ V ' 

where X is the maximum of (J) . . . over all possible choices of y, 

itj's and Vi's, such that £7 + J2i u i — m > V + J2i Vi — n, < Ui < m/7, and 
< Vi < Ui. The bounding methods of [10] are too crude for our purposes, so 
wc first need to develop better bounding techniques. 

Lemma 6 (Stirling's approximation). For n > 1, we have 

at- n\ 
V2?r < — ; ~ < e. 
yjnyn/ e) n 

Lemma 7. For values u and v, such that < v < u/2, we have 

i< <2 <£. 

' _L lay f-H-Y" 5 

yjv V V J \ u—v J 

Proof. We start by estimating the value of u\/{u—v)\. To do so, we first show that 
the sequence a n — n\/(n/e) n is increasing and b n — nl/(n/e) n+1 is decreasing 
for integers n, n > 0. Consider 

On+i = (n+l)!e" +1 n" e = i_„i g( i + i/ n) 

o„ (n+l)" +1 n!e" (l + £)" 

since lg(l + x) < x for x > —1. In a similar fashion, consider 
b n+1 (n+ l)!e"+ 2 n"+! _ ^ 1 _ e i + („ + i)i g( i-i/(„ + i)) < x 



(n + 1)™+ 2 n!e™ +1 V «■ + 1 
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since lg(l — x) < — x for x < 1. Since a u /a u - v > 1 and b u /b u - v < 1, we have 



(u-u)! " (( u - v)/e) u - v Ve/ U-n 



(u-v)\ b u - v (u-v)^ "+ 1 u - v {(u - v) I e) u ~ v u-v\e 

We divide both of these inequalities by v\, and use Lemma 6. We obtain 

1 1 /e\ v fu\ v ( u \ u ~ v / u \ 1 1 u /e\« f u \ v ( u 
ey/v\vJ Ve/ \u — vj \v J \/2n ^/v u — v \v J Ve/ \u— v 

By the precondition of the lemma, v < u/2, so that u/(u — v) < 2. Also, since 
V2tt > 5/2, the statement of the lemma follows. □ 

Let us define = min m and = min Vi . In the case where Vi 's are of the 
same order of magnitude, we can use the following lemma. 

Lemma 8. If u* > 2 and v* > 1, then ft. (£) < (^)2-(7/2)ig^-o.3 7 +(ig 



y)/ 2 _ 

Proof. To bound each individual multiplier, we apply the right part of the in- 
equality of Lemma 7. If < u,/2, then 



n(:H(^)(S 



The case where «j > m,/2 can be reduced to the case t>j = Uj/2. Next, we apply 
the inequality between arithmetic and geometric means for the values 



— — , — — — -)■•■) — i and obtain 
vi vi v 2 v 2 v y 
- .. " s .. ' ^_ L/ 



Ui times t) 2 times „ times 



TT ( < f— ^ — i = Ttt^ . Similarly, we obtain 



n 



< 



K Ui-ViJ \l>2i u i- v i) \U -V 

Finally, we apply the left part of the inequality of Lemma 7, 

n(:)<^(j;)n5^^- W2 " , "- w * , ' ,/2 (v)- 



□ 
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To bound the product f(vi, V2, ■ ■ ■ , u 7 ) :— f\ i ("*), we maximize it over all 
possible Uj's with Uj's fixed, subject to the constraint that the sum of Vi's is V. 
We say that a tuple (ui, V2, ■ ■ ■ , u 7 ) is a focaZ maximum if we can not increase 
the value of / by changing some Vi to Uj + 1 and some other Uj to Uj — 1. The 
following lemma characterizes the local maxima. 

Lemma 9. At a local maximum, 

Vj + 1 



> 



Uj + 1 ~~ Ui + 1 

is satisfied for each pair i 7^ j. 

Proof. At a local maximum, we have the following inequality 



> 



Ui 
Uj - 1 

(Uj 



1 



Dividing both parts by ( „. )!( „/-Vi)' > we obtain 

1 1 1 1 

Wi (Uj - Uj) ~~ (Uj — Vj + 1) (Vj + 1) 

From this, we get UiVj + Ui — + t>j + 1 > UjUj and the lemma follows. 
Corollary 1. a /oca? maximum, for all i, I < i < 7, we ftawe 

2 



< 



Proof. Fix i 7^ and apply Lemma 9 for the pair and for the pair (j, i). It 
follows that 



+ ■ 



> 



> 



1 



Uj + 1 Uj + 1 + 1 Uj + 1 iti + 1 
Since Uj and Uj are at least u*, we have 



1 



Since i>j/uj and Vj/uj are at most 1, 



1 



< 



1 



1 



u* + 1 



< 



u* 



Uj + 1 

Therefore, 
Finally, we observe that 





( Ui 


_ 






Vi/Ui 


Vj/Uj 


Uj + lJ 


\Ui 








Ui + 1 


Uj + 1 




Vi 




< 


2 








Ui 


Uj 


u* 







1 

< — 

u* 



min < — ,...,— >< — < max 



u~, 



V 



U 



\ v\ v 1 
I Ui u 7 



and the corollary follows. 



□ 
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Lemma 10. If u*V/U > 3, then ft. (£) < (^) 2 -(7/2) ig(«.v/c/)-o.3 7 +(i g y)/2_ 

Proof. We first maximize X with respect to v^s for fixed Uj's. At a local maxi- 
mum, Corollary 1 gives us the bound 

v % V 2 V _2V _ V 
~u i > U '~ ' U~ 3f7 ~ W 1 

so that Vi > UiV/(3U). Hence, we can apply Lemma 8 with = UiV/{3U) > 
u*V(3U) > 1. The result follows. □ 



3.1 Density-Sensitive Rank Index 

(Theorem 2 for the ranki operation). 

Proof. Let us define k := 771/7 to be the length of a block. We combine con- 
secutive blocks into larger superblocks, such that the number of unprobed bits 
in the i-th supcrblock, u*, is between k and 2k (except, possibly, for the last 
supcrblock). This can be done in a greedy fashion, considering blocks from left 
to right: we keep adding blocks to a superblock until the number of unprobed 
bits in it reaches k, at which point we finalize it and start a new one. We will 
never overshoot the value 2k, since all Mj's are at most k. It was shown in [10] 
that the number of superblocks 7 S = ©(7), and J\ i (" ; ) < J\ i ("j), where v* is 
the number of unprobed 1-bits in the i-th superblock. 

First, consider the case tn > m. Let us choose 7 to be m/(3t). We can apply 
Lemma 10 to Ili ("*)' smce ^min{i7,min{'u*}}/777 = min{n/3, kn/m} = 
min{n/3, 3tn/m} > 3. Recalling that U = m — tj and V — n — y, so that 
TO<(t)>we obtain 

C* < n ^{^j fl ^ n2 r 2-^^ l s(^/ m )-o. 3 ^ S n)/2^y HenC6j 

r > (7 s /2)lg(3tn/777) + 0.3 7s - 3(Ign)/2 = Q{{m/t) \g{{nt)/m)). 

If cm < tn < m for some positive constant c, then pick 7 = n/3. We have, 
nmin{£7, k}/m > min{cn/3, 3} > 3, and obtain 

C* < n2 r (^j fj < n2 r 2-^l 2 ^- a ^+^ n ^ 2 {^, and 

r > (7a/2)Ig3 + 0.3 7s - 3(Ign)/2 - Q{n). 

Finally, if nt = 0(777), then we pick 7 = \J nm/t. 

We bound the product ft. (£) < IX < ( m hV < C"y /7 ) < (T?) 
directly using simple inequalities (u/v) v < (") < 77". 

Thus, 
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since \Jnmt = uj(nt) > 2n. 
We can bound r by 




□ 

3.2 Density-Sensitive Select Index 

(Theorem 2 for the selecti operation). 

Proof. As with the proof for the case of ranki operation, we consider three cases: 
where nt = to(m), nt = &(m), and where nt = o(m). We simulate the set of 
queries 

Q* = {"query selecti B (l,ifc)"| 1 < i < 7}, 

where k = [n/j\. Accordingly, we split bit vectors B into 7 blocks of equal 
cardinalities n\ = n% = . . . = n 7 = k. The i-th block starts at position 
selecti ((« — l)fc) + l and ends at position selecti (ik), so that the cardinality of each 
block (the number of 1-bits in it) is exactly k (recall that we defined selecti (0) = 
for convenience). We set % = {B e {0, 1}™| number of 1-bits in B is n}. We 
choose parameter 7 depending on the relationship between nt and m. In the 
case where nt — uj(m), we will choose 7 = m/(3t). For the case mt = 0(n), 
we need an additional requirement that 7 < n/3, so that we will choose 7 = 
min{m/(3£), n/3}, we will clarify this requirement later in the proof. Finally, for 
the case nt = o(m), we will choose 7 = n. Note that in all cases, the number 
of unprobed bits U is m — t-f > 2m/3, so that the average number of unprobed 
bits per block is at least (2/3)n/j (we expect most of the blocks to have at least 
constant fraction of unprobed bits) . 

We define superblocks as follows. The i-th superblock (except, perhaps, for the 
last one) will contain consecutive blocks z,_i + 1, . . . , z i} such that the number 
of unprobed 1-bits in the i-th superblock 

v i = V Z i-!+l + V Zi _ 1 + 2 + ...+V Zz 

satisfies k < v* < 2k. Note that this is always possible, since Vi < ni = k. 
And hence, 7 S , the number of superblocks, is at least V/(2k). The number of 
unprobed bits of i-th superblock is given by 

K = u Zi _ 1+x + u Zi _ 1+2 + . . . + u Zz 

We use inequality 
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to bound the number of bit vectors compatible with a given leaf. The total 
number of bit vectors compatible with all the leaves is then 

*-* g (XO (;■)■-■(:•) 

We can derive a bound on P, P < n2 r X, where X is the biggest product of 
binomial coefficients in this sum. To derive a bound on X, we can, for example, 
use Lemma 8. A difference with the proof of the ranki case is that we do not 
need to "redistribute" the weight of V between v^s uniformly as it was done in 
Lemma 9 and Corollary 1, since we have bounds k < v* < 2k already. To derive 
a lower bound for r, we observe that J2 X \C(x)\ = = (™) < P. Therefore, 



r > lg ^ - 1: 




First, we consider the case where nt/m = fi(l). Recall that we chose the 
parameter 7 = min{m/(3i), n/3} = 0(m/t). The goal is to derive an upper 
bound on 



subject to constraints 



y + v * 1 = n > and 



m 

H< 3 

Since there is a bound on v*'s, namely, v* > k, it seems that we can apply 
Lemma 8 directly and obtain a bound on X. The caveat is that, if V is too 
small, then the number of superblocks 7 S is small as well, and the bound will 
turn out to be weak. This problem did not arise in the proof of the ranki 
case, since the bound on j s was based on the fact that J2 u i — 2n/3, and 
we were grouping blocks into superblocks based on values of Uj. However, 
in this proof, we form superblocks based on Vi's, so that we need to bound 
their sum, V = J2i v i> horn below. 

For this purpose, we use the idea that is similar to an idea in the proof of 
Lemma 10. Let us vary u^s and v^s in order to maximize 

:)n(: 

As a very rough estimation, we can state the following: since t-f < m/3, we 
expect that y will be at most n/3, and so that V — n — y > 2n/3, which is 
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sufficient for our purposes. More formally, Lemma 9 gives us the following 
conditions at a local maximum: 

Vi + 1 > y. > v 



u, + 1 ~ tj + 1 ~ m/3 + 1 
Thus, for any i € [7], we have 

(y + l) («i + l)>y(«i + l). 
Summing them up, we obtain 

V > y ' - 7 > n - y /o T 1 - o = T - 2V ' 
m/3 + 1 m/3 +1 3 3 

since 2 < 7 < n/3. Thus, 

- 9 

Now, it is easy to derive a bound on the number of superblocks, j s , 

V 5n 7 57 /m 

7 S > — > > — = ( — 

' 2fc ~ 18 n ~ 18 V t 

Let us apply Lemma 8 to (3). We obtain, 

X < ^W (7s/2)lgfc - e(7s) ^ < 2-(^/2)igfc-©( 7s )( TO 



<- 2-6 , (™/*) lgfc-@(m/t)+e(lgn) 



So that, in the case where nt = oj(m), we obtain 

r = e(^(-))-o(±). 



since k = n/7 = 3nt/m. In the case where nt = 0(m), wc obtain 

r = 0(n), 

since k = n/7 > 3, fc = 0(1). 

It remains to consider the case where mt — o(n) . Recall that we chose 7 = n 
in this case, so that we select all the 1-bits in the bit vector using our queries. 
Hence, the number of compatible bit vectors |C(x)| with any leaf x is exactly 
1. We can bound the sum 

( n ) \ n / 



1G 



here we used the fact that (") < (eu/v) v . Thus, 

r = 6>(nlg^) -0(n) 
This completes the proof of Theorem 2 for selecti operation. □ 



4 Conclusions 

We have provided matching density-sensitive upper and lower bounds on the 
redundancy required for rank and select operations on sparse bit vectors in the 
succinct index model. These results improve signficantly on known results for 
the case where n = o(m). Although our results depend on both m and n, we do 
not take into account any regularities in the distribution of Is in the bit vector; 
an interesting direction of research would be to find appropriate measures of the 
regularities of Is in the bit vector that could lead to further reductions in the 
index size. 
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